Balancing of neural contributions for multi-modal hidden state association
نویسندگان
چکیده
We generalize the formulation of associative reservoir computing networks to multiple input modalities and demonstrate applications in image and audio processing scenarios. Robust association with reservoir networks requires to cope with potential error amplification of output feedback dynamics and to handle differently sized input and output modalities. We propose a dendritic neuron model in combination with a modified reservoir regularization technique to address both issues.
منابع مشابه
Incorporating Global Visual Features into Attention-based Neural Machine Translation
We introduce multi-modal, attentionbased Neural Machine Translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convolutional neural network and are incorporated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to init...
متن کاملExperimental and finite-element free vibration analysis and artificial neural network based on multi-crack diagnosis of non-uniform cross-section beam
Crack identification is a very important issue in mechanical systems, because it is a damage that if develops may cause catastrophic failure. In the first part of this research, modal analysis of a multi-cracked variable cross-section beam is done using finite element method. Then, the obtained results are validated usingthe results of experimental modal analysis tests. In the next part, a nove...
متن کاملMulti-modal Particle Filtering for Hybrid Systems with Autonomous Mode Transitions
Model-based diagnosis of embedded systems relies on the ability to estimate their hybrid state from noisy observations. This task is especially challenging for systems with many state variables and autonomous transitions. We propose a fair sampling algorithm that combines Rao-Blackwellised particle filters with a multi-modal Gaussian representation. In order to handle autonomous transitions, we...
متن کاملComparative analysis of hidden Markov models for multi-modal dialogue scene indexing
A class of audio-visual content is segmented into dialogue scenes using the state transitions of a novel hidden Markov model (HMM). Each shot is classi ed using both audio track and visual content to determine the state/scene transitions of the model. After simulations with circular and left-to-right HMM topologies, it is observed that both are performing very good with multi-modal inputs. More...
متن کاملMulti-directional continuous association with input-driven neural dynamics
We present an input-driven dynamical system approach to continuous association. Previous formulations of associative reservoir computing networks and associative extreme learning machines are unified and generalized to multiple modalities. Association in these networks proceeds by externally driving parts of the network. Through continuous variation of driving inputs, a continuous association o...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2012